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Humans are usually accurate whern estimating Ineading or path from optic flow, even in the 
presence of independently moving objects (IMOs) in an otherwise rigid scene. To invoke 
significant biases in perceived heading, IMOs have to be large and obscure the focus of 
expansion (FOE) in the image plane, which is the point of approach. For the estimation 
of path during curvilinear self-motion no significant biases were found in the presence 
of IMOs. What makes humans robust in their estimation of heading or path using optic 
flow? We derive analytical models of optic flow for linear and curvilinear self-motion 
using geometric scene models. Heading biases of a linear least squares method, which 
builds upon these analytical models, are large, larger than those reported for humans. 
This motivated us to study segmentation cues that are available from optic flow. 
We derive models of accretion/deletion, expansion/contraction, acceleration/deceleration, 
local spatial curvature, and local temporal curvature, to be used as cues to segment 
an IMO from the background. Integrating these segmentation cues into our method of 
estimating heading or path now explains human psychophysical data and extends, as well 
as unifies, previous investigations. Our analysis suggests that various cues available from 
optic flow help to segment IMOs and, thus, make humans' heading and path perception 
robust in the presence of such IMOs. 
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INTRODUCTION 

Optic flow, the apparent change of structured patterns of hght 
on the retina during head, eye, or body motions, contains cues 
about these motions and the viewed environment. For instance, 
optic flow contains information about the 3D linear velocity and 
the 3D rotational velocities, e.g., the yaw, pitch, and roll velocity, 
of the eyeball (Longuet-Higgins and Prazdny, 1980). In addi- 
tion, optic flow includes motion parallax cues that, e.g., provide a 
relative-depth order between foreground objects and background 
(Helmholtz, 1925; Gibson et al., 1995). Self-motion in a rigid 
environment results in optic flow that is entirely defined by 3D 
linear and 3D rotational velocity parameters and depths as seen 
from the observer's point of view. In practical situations the envi- 
ronment is non-rigid, e.g., other people walk within the visual 
field. Flow generated by these moving people is inconsistent with 
the parameters of self-motion. Instead, this flow is a superposition 
of the self-motion and other people's motion. Therefore, our goal 
is to analyze segmentation cues and their possible effect on the 
estimation of self-motion by integration of flow from only rigid 
parts in the environment. 

Our work is motivated by three studies. Two studies measure 
a heading bias for linear motion toward fronto-parallel planes 
where one plane moves independently, the independently moving 
object (IMO), and another serves as rigid background (Warren 
and Saunders, 1995; Royden and Hildreth, 1996). The third study 



measures accuracy of path perception for curvilinear motion 
above a ground plane where IMOs are modeled as cubes (Fajen 
and Kim, 2002; see Warren and Mestre, 1991 for path perception 
without IMOs). In the first two studies heading biases are present 
either if the IMO obscures the focus of expansion (FOE) of the 
background motion or if the IMO's path crosses the observer's 
path. The FOE is a point in the image plane that is introduced 
by linear motion and appears at the intersection of the head- 
ing vector, or linear velocity vector, with the image plane. The 
third study did not find any significant differences in path per- 
ception whether IMO's had been present or not while changing 
their position, path, or degree of transparency. These seemingly 
contradictory results from these studies raise the question: what 
are the differences in task and configuration in the experiments, 
and to what extent do these explain the heading bias or absence 
of error in path perception? Furthermore, we ask: what are the 
mechanisms that enable humans to robustly perceive heading or 
path under these circumstances? 

Assume an observer is on a curvilinear path while an IMO is 
intersecting that path, like in Figure lA. This observer registers 
the optic flow as shown in Figure IB. We propose flow-based cues 
for segmenting such an IMO. The first cue is accretion/deletion, 
see Figure IC. This cue is present due to a faster moving object, 
which passes in front of a slower background. One edge of the 
object uncovers background, the accretion (bright), while the 
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Other edge covers or occludes background, the deletion (dark). 
Such cues are present at depth discontinuities and between 
IMOs and background. As a second cue we define expan- 
sion/contraction, see Figure ID. Spatial derivatives in the neigh- 
borhood of these transitions in the image plane are summed. 
Expansion indicates a source in the flow field (bright), and con- 
traction indicates a sink in the flow field (dark). The third cue 
is acceleration/deceleration, which describes the temporal char- 
acteristics of the flow at one spatial location in the image, see 
Figure IE. Acceleration occurs if pixel-motion increases at the 
next time frame (bright) and a deceleration if pixel-motion 
decreases (dark). We define local, spatial curvature of flow fields 
as fourth cue, see Figure IF. This cue is a local approximation 
of the curvature of an integral curve (a solution) interpreting the 
equation that describes optic flow as a system of non-linear ordi- 
nary equations (ODEs). Since this curve exists in the image plane 
we call it spatial, and because we use an approximation that is 



good only in the vicinity of the considered point, we call it local. 
Curvature appears either concave (bright) or convex (dark). As a 
fifth segmentation cue we define the local temporal curvature of 
the flow field, see Figure IG. Unlike the spatial curvature, this cue 
describes the temporal change for a single temporally fixed loca- 
tion in the image. The curve is defined by the temporal change of 
the vector in that location. 

Our goal is to derive analytical models for motion integration 
and segmentation when IMOs are present in the environment. We 
develop a linear model for the estimation of self-motion assuming 
linear motion and for the estimation of radius assuming curvi- 
linear motion. Two integration mechanisms are proposed. One 
integrates over the entire visual field and another segments the 
IMO and integrates only flow from the rigid background. To 
establish such segmentation, we derive analytical models for the 
above mentioned segmentation cues (see Figures IC-G). These 
cues are integrated into estimation methods of heading and path 
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FIGURE 1 I Shows the problem of heading estimation in the 
presence of an IMO together with available flow-based 
segmentation cues. (A) Sketcli of the movements. (B) Detected optic 
flow (Zach at al., 2007). (C) Accretion {>0) and deletion (<0) cue. 



(D) Expansion (>0) and contraction (<0) cue. (E) Acceleration (>0) 
and deceleration (<0) cue. (F) Local, spatial curvature cue. (G) Local, 
temporal curvature cue. The sign indicates concave (>0) or convex 
(<0) curvature. 
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as a possible solution for the combined motion integration and 
segmentation. 

We continue this article by describing the Methods, a descrip- 
tion of an analytical model for optic flow, called the model 
of visual image motion, for different scene and motion con- 
figurations. This model is used to define and derive segmen- 
tation cues and methods estimating heading or path. Results 
show quantitative and qualitative evaluations of these derived 
models by plugging in the settings as used in the targeted psy- 
chophysical studies (Warren and Saunders, 1995; Royden and 
Hildreth, 1996; Fajen and Kim, 2002). The Discussion provides 
a classification of our analytical models in terms of existing 
work on heading and path perception both in modeling and in 
psychophysics. 

METHODS 

This section is organized into three parts. In the first part, 
we derive a motion model for linear motion toward a plane 
and curvilinear motion above a ground plane or toward an 
initially fronto-parallel plane. The second part describes meth- 
ods that estimate the horizontal position of the FOE for linear 
motion toward a plane or the radius from curvilinear motion 
above a ground plane. In the third part, we define analytical 
models for segmentation cues of stationary and independentiy 
moving objects. Table 1 gives a summary of identifiers used in 
the derivations and model. 

In some cases we define image positions, lengths, or the x/y- 
components of the linear velocity in degrees of visual angle. 
For positions this translates into XMeter =/ • tan(xAngie) and for 
lengths /Meter = 2 ■/■ tan( /Angle /2), where / denotes the focal 
length of a pinhole camera model. Velocities are transformed 
according to VMeter/s = • tan(vAngie)> where [Vz] = m/s is the 
velocity along the optical axis of the camera. Thus, we can specify 
positions and velocities either in degrees or degrees per second, or 
in meters or meters per second. 

PROJECTION OF MOVING 3D POINTS ONTO A 2D IMAGE SURFACE 
USING A 3D MOTION MODEL 

We follow Longuet-Higgins and Prazdny's (1980) derivation of an 
equation for visual image motion, assuming a rigid environment, 
an instantaneous 3D motion, and a pinhole camera model. This 
derivation is briefly repeated to give an idea how our model fits 
and extends the model of Longuet-Higgins and Prazdny (1980). 

We assume that 3D points are defined by {X, Y, Z) while 
the z-component is measured along the optical axis, the 
x-component to the left, and the y-component to the top using 
a right-handed coordinate system. We use uppercase letters to 
denote points or their components that are sampled in 3D. 
Three-dimensional points are projected onto the image plane of 
a pinhole camera. This image plane is at the distance / in front 
of the nodal point. Points projected onto this image plane are 
{x,y,f) =f/Z- {X, Y,Z) and we denote them by lower-case 
letters. We assume the 3D motion as a combination of a 3D 
translational velocity vector v = {Vx,Vy, v^)' and a 3D rotational 
velocity vector m = (wx, (Hy, (HzY excluding higher order tem- 
poral changes, like accelerations. We use the super-script t to 
denote the vector-transpose. According to classical kinematics 



Table 1 | Lists all identifiers with a brief description and their physical 
units. 



Identifier 


Description 


Unit 


t 


Time 


s 


f 


Focal length of a pinhole camera 


m 


(X, y, f) 


Point on the image plane from a pinhole 
camera nodal point 


m 


Z(x. y. f) 


Distance measured along the optical axis 
at (X, /) and t 


m 


V = (Vx. Vy, Vz) 


3D linear velocity vector 


m/s 


O) — (oox, Wj/, (xiz) 


3D rotational velocity vector (pitch, yaw, 
roll) 


°/s 


(X, /) 


Model of visual image motion 


m/s 


p 


Angle of edge normal pointing outward of 
the object 


° 


n (t) — (n-^ rtw 

II \i-f — \^''X' ''yi ''Zf 


Normal vector of a pictured plane 


None 


d(t) 


Distance of a pictured plane 


m 


h 


Height of the camera above ground 


m 


O) — iUy 


Rotational yaw-velocity 


7s 


V 


Linear velocity along the optical axis 


m/s 


r 


Radius of curvilinear motion path 


m 


to 


Time-to-contact with a pictured plane 


s 


(11. i;) 


Image angle coordinates of the focus of 
expansion 


none 


A 


Accretion (A > 0) and deletion (A < 0) 


m/s 


8 


Expansion (5 > 0) and contraction (8 < 0) 


1/s 


a 


Acceleration (a > 0) and deceleration 
(a < 0) 


m/s^ 


1} 


Local, spatial curvature: Concave (i} > 0) 
and convex {a < 0) 


1/m 


Y 


Local, temporal curvature: Concave (y > 0) 
and convex (y < 0) 


1/m 


(Xl, Yl) 


Position of the independently moving 
object in the image 


m 


{wi, hi) 


Width and height of the object in the 
image 


m 


(wb, ha) 


Width and height of the image plane of 
the pinhole camera 


m 



If vectors appear in the first column, we denote the unit of each component. 
Some experiments indicate speeds in m/frames rather than m/s, which can be 
converted using the frame rate. 
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these two velocities define the motion of a 3D point velocity by 
(X, Y, Z) = —{Vx, Vy, Vz) — (cOx, Wy, coz) X {X, Y, Z) assuming 
(0, 0, 0) is the center of rotation and x denotes the cross-product 
(Goldstein et al., 2001). Computing the temporal derivative of 
the projected points in the image plane and plugging in the 
definition of 3D point velocity yields the visual motion equation 
(Longuet-Higgins and Prazdny, 1980): 



+ 




xy 

f\(f'+/) 



—xy 




(1) 



This Equation ( 1 ) expresses the image velocity (x, y) ' of 3D points 
{X, Y, Zy that are projected onto the location {x, yY in the image 
plane. In Equation ( 1 ) we dropped the third component because 
its value is constantly zero. 

The model of visual image motion as defined in Equation (1) 
has several interesting properties. First, image motion induced 
by linear and rotational velocity superimpose linearly. Second, 
only the image motion induced by linear velocity depends on the 
distance Z of the sample point. The speed of this flow is inversely 
proportional to depth, which means that points further away have 
a lower speed than points that are close. A third property is the 
symmetry between the x- and y-components of linear and rota- 
tional velocity and their induced image motion. For instance, 
when translating to the left the same image motion is gener- 
ated if translating upward and flipping x and y components of 
the resulting image motion. The specific location (r\,t,) = (v^/Vz, 
Vy/Vz) denotes the FOE. The location (wx/coz, My/coz) is called 
the center of rotation, or COR (Gibson, 1950; Longuet-Higgins 
and Prazdny, 1980). 

The computation of visual motion with Equation ( 1 ) requires 
the distance Z of surfaces from the camera expressed in the cam- 
era's coordinate frame. Thus, the distance Z is a function of the 
position in the image plane (x, y, f) as well as of the time f. It can 
be interpreted as the surface function Z{x, y, t) as seen through 
the pinhole camera at time t. The next paragraphs use Equation 
(1) either to further specify the surface function Z or to further 
restrict the possible paths of self-motion or object motion. This 
will yield the three model instances: ( 1 ) the linear motion of an 
observer toward a plane, (2) the curvilinear motion of an observer 
toward a fronto-paraUel plane, and (3) the curvilinear motion of 
an observer above a ground-plane. 

For the first motion model, that of linear motion toward a 
plane, we assume that the plane is defined in Hessian normal 
form with the normal vector n = {n^, riy, n^) and distance d from 
the origin, here the nodal point of the pinhole camera. Figure 2A 
shows an example of such a motion and plane configuration. We 
assume that the plane is stationary in 3D space. Then, the viewer- 
dependent distance of the plane changes with time according to 



In Equation (2) do denotes the initial distance at f = Os. The 
distance d{t) changes by the distance t ■ v traveled along the 
normal vector n. This is formalized by projecting t ■ v onto 
the normal vector n (calculating the inner vector product). 
The Hessian normal form for such a time variant plane reads 
(Hx, iiy, itzYiX, Y, Z) — d(t) = 0 using an arbitrary 3D sample 
point {X, Y, Z). Plugging this equation into the projection for 
a pinhole camera and solving for Z as a function of x, y, 
and f gives: 



Z(x, y, t) 



f-do-f-t-(Vx-nx + Vyny + Vz- «z) 



x-Hx+y-Uy+f ■ 



■f 



io — t ■ V n 



(3) 



with the lower-case p = (x, y, f) being a location in the image. 
We use Equation (3) and plug it into Equation ( 1 ) setting all rota- 
tional velocities to zero. This gives the model instance for linear 
motion toward a plane: 



X- nx + y ■ Hy + f ■ i 



yj f-do-f-t-iVx-nx + Vy-ny + Vz-nz) 



-f 0 X 
0 -/ y 




(4) 



By plugging in the time-dependent surface function Z, we intro- 
duce a time dependency that was not explicitly stated before. 
In Equation (1) the only time-dependent variable is the sur- 
face function Z, since the focal length / is constant (no zoom), 
the sample locations {x, y f) on the image plane are con- 
stant (no lens jitter or shift of the nodal point), and the linear 
and rotational velocities are constant (no acceleration). Studying 
Equation (4) further shows that it is, in general, a polynomial 
in X and y, and non-linear in the linear velocity components 
Vx, Vy, and Vz- 

For the second model instance, we assume curvilinear self- 
motion or object motion along a circular path with translational 
velocity v = (0, 0, m ■ r)' being tangent to that path and the 
rotational velocity m = (0, to, 0)^ In this case the observer tra- 
jectory is on an orbit that is centered at an arbitrary reference 
in space. In addition, we assume traveling toward an initially 
fronto-paraUel plane that appears at the distance d^. Figure 2B 
shows the geometry of this configuration. The normal vector of 
the plane changes with time and is n(f) = (— sin(a) ■t),0, cos(a) • 
f))^ and so does the distance, which is d{t) = do — r ■ sin(M ■ t). 
We use the Hessian normal form for a plane to plug in the above 
values combined with the perspective projection function and 
solve for Z. This result for Z is then plugged into Equation ( 1 ) 
which yields: 



x\ X- sin(a> • t) + f ■ cos{(S) ■ t) 
y) f ■ (do - r ■ sm{(D ■ t)) 



d{t) = do - t ■ (Vx- Ux + Vy ■ Hy + Vz ■ Hz) = do 



t ■ v'n. (2) 



/ 



xy 



(5) 
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front-view 



A Straight patli 



top-view 




C Independently moving object 
+1/2, 



-1/2 



background 



IMO 



(Z,, V,, m,) 



-1/2 



0 



+1/2 



V 



B Curvilinear path 
front-view top-view plane 




D Integration along contour 
+ 1/2 



rsin(6) -1/2 



background 



/?=90°4. 









^=180^ 


IMO 

r 



/?=270' 



-1/2 



+1/2 



FIGURE 2 I Shows motion/scene configurations and thie definitions of 
IIVIOs. (A) A straight, linear path toward a plane; here, we show a top-down 
view. (B) Configuration for a curvilinear path toward an initially (t = Os) 
fronto-parallel plane, again as top-view. (C) Shows the definition of an IMO in 



the image plane by its position (x/, yi), size (w/, hi), and additional parameters 
(Z/, vi, wi). The background has the scene and motion parameters 
(Zg, Vb, mg). (D) Integration along the contour happens using normal vectors 
with direction p and following the black arrows drawn on the contour 



This Equation (5) depends on the angular velocity co and the 
radius r of the circular motion path. These two variables define 
the curvilinear motion. The image velocity depends on these 
variables in Equation (5) non-linearly. 

For the third model instance we consider again curvilinear 
motion but change the scene geometry from a fronto-paraUel 
plane to a ground-plane. The camera is assumed to be at the 
distance d = h above the ground, which has the normal vec- 
tor « = (0, 1, 0)^ In this case, the distance and normal vector 
do not change with time, which makes the ground-plane sur- 
face function and, thus, image velocities time-constant. Plugging 
these definitions into Equation (3) and plugging in the result into 
Equation (1) together with the definition of curvilinear motion 
yields: 



y r 



f 



xy 



(6) 



Table 2 summarizes these three model instances that we use to 
estimate linear motion or the radius of curvilinear motion. 

ESTIMATE THE HORIZONTAL FOE POSITION FOR LINEAR MOTION OR 
RADIUS FOR CURVILINEAR MOTION 

This subsection contains two derivations: the first is the deriva- 
tion of a linear least squares method to estimate the horizontal 



FOE position (variable ri), and the second is the derivation of a 
method to estimate the radius r for curvilinear motion. We use 
the derived estimates to evaluate their robustness in case of the 
presence of an IMO. This IMO is parameterized by its position, 
size, and 3D motion relative to a stationary camera. Figure 2C 
depicts these IMO parameters. 

Estimation oftiorizontal FOE 

We start the derivation using Equation (4) assuming that back- 
ground and IMO are fronto-parallel planes, setting the normal 
vector to M = (0, 0, 1)*^ and putting into the denominator on 
the right-hand side of the equation. We name the resulting ratios: 
= do/Vz the time-to-contact, ti = v^/Vz the horizontal image 
position of the FOE, and ^ = v^/v^ the vertical image position of 
the FOE. We use the resulting model equation in a linear least 
squares approach to optimize for r\ and t;. 



F(,.0 = JJ 



dx dy with 



tr - t 



(7) 



This model in Equation (7) expresses the squared distance of each 
individual residual vector in the image plane. The residual is com- 
puted between the given optic flow (x, yj and a model thereof. 
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Table 2 | Lists models of visual image motion. 



Motion 



Scene 



Model Instance 



General 

V = (Vx, Vy. Vz) 
to — ((jOx, OOj/, OOz) 

Translation 

V= (Vx, Vy. Vz) 

(u = (0,0,0) 

Curvilinear 

= (0, 0, M ■ r) 
<S = (0, u), 0) 



General 
Z(x, y, t) 

Plane 

n = (Ox, Hy. Hz) 

d(t = 0) = do 

Fronto-parallel plane 
n(f = 0) = (0,0, 1) 
d{t = 0) = do 




1 / Xy -(f2 _,_ ^2) fy \ 

" I 7 I (f2 + ).2) -xy -fx. 



X ■ nx + y ■ ny + f ■ Hz I - f 0 x 

f ■ do - f ■ f ■ (Vx ■ Ox + ■ /ly + Vz ■ "z) \ 0 -f / 



X \ X ■ sin((o ■ t) + f ■ cos(a) ■ f) / X \ CO 

■ — 'r '< ■ ^ ■ ^ 

f ■ (do — r ■ sin(a) ■ f)) 



xy 



iJiy 



Curvilinear 
v = (0, 0, 00 ■ r) 
m= (0,u), 0) 



Ground plane 
n(t) = (0,1, 0) 
d(f) = h 



y r 
IT," 



'f2+x2 

f \ xy 



We denote the motion and scene geometry as seen through a pinhole camera model. For the translation toward a plane the normal vector is Independent of time; 
in contrast, for curvilinear motion the normal changes w/t/i time. For motion parallel to a ground-plane, normal and distance stay constant over time. 



which contains the parameters r\ and t,. All residual vectors are 
integrated over the image plane, which is indicated by the double 
integral. We assume that the reciprocal y of the temporal dis- 
tance toward the wall is constant for all locations in the image 
plane. This assumption actually does not hold true in the presence 
of IMOs, which will introduce a bias. Furthermore, we assume 
that Y is known, e.g., estimated from the binocular disparity c of 
the background by y{t = 0) = —c/c for c = • f /Z with b being 
the inter-camera distance of the stereo setup. Later, we relax this 
assumption by instead assuming that there is a segmentation cue 
available that indicates the image region depicting the IMO and 
the image region depicting the background. To compute the solu- 
tion for Equation (7) we take the partial derivatives of F with 
respect to t) and t,, set these to zero and solve for y\ and t,, which 
gives: 



Ti = 



JJx — x/ydx dy 



and i, ■■ 



JJ y — y/ydx dy 
f-jjldxdy 



(8) 



The solution is indicated using a hat-symbol. The solution for fi 
depends on x, x,f, and y, and neither on y nor y. Dependencies for 
i, are similar concerning the vertical a5ds. In the following steps we 
restrict the derivation to the horizontal component fi, noting that 
the derivation for the i, component is similar due to its symmetry. 
For the background and IMO motion we use the following model 
instances: 



VB/I / \-f -Vy.B/I + y- Vz,B/I , 

1 

[^B/I = -1 ■ 

ao.B/I — t ■ Vz^B/I 



with 



(9) 



Equation (9) defines the visual motion for the background assum- 
ing self-motion and for the IMO composed of self-motion and 
object motion. The subscript B indicates the visual motion and 



parameters for the background and the subscript I the visual 
motion and parameters of the IMO. 

To define the registered analytical optic flow we use the two 
model instances from Equation (9) that define the flow for the 
IMO region and background in the image plane. The linear 
method estimating the horizontal FOE position, see Equation (8), 
integrates all flows within the image plane. Per definition of the 
analytical flow, we split this integration into two areas one for the 
IMO and one for the remaining background. These areas for IMO 
and background are defined by: 

Ai = [xj - wi/2, XI + wi/2] X [yi - hi/2, yi + hi/l] (10a) 
Ab = [-wb/2, wb/2] X [-hB/2, hB/2] \Ai. (10b) 

The size of the image plane is wb x Hb and the size of the IMO 
is wi X hi. We weigh the IMO area by the segmentation signal 
(1-s) to discount the IMO's flow within integration. To simplify 
the derivation, we use the auxiliaries h and I2 for background 
and IMO motion as well as their respective areas: 

Ii = X — x/y dx dy and I2 = f ■ 1 dx dy. (11) 

The dependency on background or IMO motion always coin- 
cides with the respective motions and, thus, is implicitly con- 
tained in the 2nd sub-index. For instance, Ii ^ is the evaluation of 
the 1st integral expression integrating background motion within 
the background area, and Ii j is the evaluation of the 1st inte- 
gral expression integrating IMO motion within the IMO area. 
Splitting the integration according to the different motions and 
summing the evaluated integrals gives: 



h.B.AB + (1 - 5) ■ -fl,/,Af 
h.B.AB + (1 - S) • h.I.AI 



(12) 
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The integrals in Equation (12) can be evaluated analytically since 
all functions are polynomials in x and y. We define the heading 
bias in horizontal, linear motion as estimate minus the back- 
ground motion: At] = fj — ^b- Plugging in all evaluated integrals, 
we get: 



ATI = fi - TlB = (1 - 5) 



wihi 



+ 



XI 



f 



M-B VxJ f [IB Vz,B 



(13) 



Intuitively, the bias is introduced because the registered, analyt- 
ical flow contains an IMO, while the model from Equation (8) 
assumes self-motion in a rigid environment. Assuming a full seg- 
mentation (s = 1), the bias vanishes. The bias term has several 
properties: it is proportional to the area of the IMO while keep- 
ing other parameters constant; and it depends on the horizontal 
position XI of the IMO, the focal length /, the ratio of reciprocal 
distance values [ii and [Lb> and also the motion components of 
IMO and background. This heading bias is without units because 
Ti is defined as the ratio of two velocity components. Furthermore, 
the bias At] is independent of the vertical position of the IMO and 
the vertical motion components of background Vy^g and IMO Vyj. 

Estimation of radius for curvilinear motion 

Our goal is to find a constraint for a linear estimation of the radius 
given the model instance of ground-plane motion [see Equation 
(6)] and to use this linear model to derive a bias term when there 
is an IMO. In the first part we derive a linear model and in the sec- 
ond part a bias term based on that model. The bias term involves 
the definition of analytical flow for the IMO, a fronto-parallel 
plane, and background. By dividing Equation (6) through the 
rotational velocity m, we get: 



1 



11 
Jh 



Iff+X^ 

f\ xy 



(14) 



To estimate r, we multiply this Equation (14) by the vector 
(— y, xj which cancels the dependency on the rotational velocity 
M. Solving Equation (14) for r yields the linear constraint we were 
looking for: 



h- 



X ■ xy - (f^ + x^) ■ y 
x ■ y^ — xy ■ y 



(15) 



With this estimate of r the path can be predicted in the image 
plane assuming that heading is tangent to the trajectory. The 
predicted path is given by the curve: 



Xcit) 

ydt) 



■-f 



l/sin(Mf) — tan(eof) 
-h/r- l/|sin((joOI 



(16) 



The absolute value in the definition of the y-component accounts 
for negative angular velocity. In the limit m • f ^ Tt/2 the y- 
component reaches the value of —h/r. This is the point of the 
circular path furthest away from its center around which the 
observer is orbiting. 



To improve the estimate of r, we separately integrate the 
numerator and denominator of Equation (16) assuming that 
these are independent measurements. Thus, we define the two 
integral expressions: 

73 = JJ" X ■ xy — (f^ + x^) ■ y dx dy and 

74 = JJ" X ■ y^ — xy ■ y dx dy. (17) 

For instances of the motion model we plug in Equation (5) for 
the IMO motion and Equation (6) for the background motion, 
adding the sub-script 7? and 7, respectively, to the path parameters 
M and r. We sum the evaluated integrals from Equation (17) as 
before in Equation (12). We define the bias for curvilinear path 
motion p,- based on the ratio between the estimated radius r to the 
radius of the background motion r^. An evaluation of all terms 
yields the bias: 



MB 
(1); 



r 



MB 



(wghl - wi {lyi - hif^ 
+ 24-(l-s)f^^gwihiyi 

(wBhl - wi {lyi - hif^ 
+ 2-{l-s)-wihi{l2y] + h]). 



(18) 



Intuitively, this bias term is introduced by assuming all flow to 
be generated by self-motion above a ground plane, see Equations 
(15) and (17). However, the IMO is neither rigidly connected 
to the background nor does it move with the same parameters 
as the background. Thus, an analytical evaluation of the inte- 
grals according to the registered flow for IMO and background 
area separately yields a deviation from the expected self-motion 
model. Properties of this bias are: It has no units; it is inde- 
pendent of the horizontal position xi of the IMO; and for a 
full segmentation it evaluates to the value one. Table 3 sum- 
marizes the two bias terms from Equation (13) to (18). Thus 
far, we have integrated motion over the entire image plane. 
Next, we introduce motion segmentation cues, deriving analyt- 
ical models for such cues. These cues can encode information 
about motion parallax, independent motion, and background 
motion. 

ANALYTICAL MODELS FOR SEGMENTATION CUES IN VISUAL MOTION 

In this subsection the goal is to derive analytical models of 
segmentation cues in order to integrate only over the back- 
ground, rather than the entire visual image plane that may 
contain regions with IMOs. Furthermore, segmentation cues 
can help to develop a qualitative understanding of the scene 
in terms of rigidity or depth order. Inspired by psychophysics 
(Gibson, 1950; Koenderink and van Doorn, 1976; van Doorn 
and Koenderink, 1982; Braddick, 1993), neurophysiology (Orban 
et al, 1992; Eifiaku and Wurtz, 1998; Orban, 2008), imaging 
studies (Zeki et al, 2003; Bartels et al, 2008), and model- 
ing work (Barnes and MingoUa, 2012), we define five distinct 
segmentation cues. 

The first cue is the accretion and deletion cue. We incor- 
porate into our definition of accretion/deletion the orientation 
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Table 3 | Lists the bias terms for translational motion toward fronto-parallel planes and curvilinear motion above a ground plane where an 
IMO is modeled as fronto-parallel plane. 

Scenario Bias term 

Translational toward fronto- Ai\ = 0-s) \^ I ~~ + "T ~ + T TT^ I with [11 = ^ and [le 

parallel planes 



WstlB \ Vz.S f V-B ^x.l f\^BVz.Bj d^j-t-V^j c/q 



Curvilinear path motion above a ^^^e / ^^3 _ (jy, - hif) + 24 ■ (1 - s)f-^^Wihiyi 
ground plane and IMO as fronto- „ ^ '"1 ^ "o 



parallel plane ^ (wghl - w, (2y, - hif^ + 2 ■ (1 - s) ■ iv,h, (12/2 _,_ /,2) 



of the local edge formed between object and background since 
locally we only notice the amount of accretion/deletion that hap- 
pens normal to that edge. We define the normal of the edge to 
point outward of the IMO and this normal is defined by the 
angle f5. To encapsulate all these described properties, we define 
accretion/deletion by: 




Equation (19) can be used by any instance of the motion model in 
Table 2 by plugging in the respective motion parameters for back- 
ground and IMO. If the IMO's motion to the right is faster than 
the background to the right, then a deletion occurs (A < 0) at the 
right, vertical edge of the IMO. On left vertical edge of the IMO 
an accretion occurs (A > 0). The unit of accretion and deletion 
is m/s. 

As a second cue we utilize the local expansion/contraction of 
the flow field. This cue is motivated by the divergence definition 
for vector fields. Consider the Jacobian matrbc J computed from 
the vector field (x{x, y, t),y{x, y, t))* by taking the partial deriva- 
tives of ic and y with respect to x andy. The expansion/contraction 
cue can then be determined from the trace of the Jacobian, 
trace(J), namely 

8 = 8; - 8b with hs/i = dxXB/i + dyh/i- (20) 

We assume that these derivatives are locally computed in the 
neighborhoods of an image location {x, y) that falls onto the 
edge between IMO and background. Then, the difference between 
the locally evaluated cue for the IMO and for the background 
is computed. If the vectors that belong to the IMO or object, in 
general, form a radially outward pointing pattern like a source 
in a vector field, and if such pattern is faster than that for the 
background, the overall cue indicates an expansion (8 > 0). An 
expansion pattern exists also, if the background has the charac- 
teristic of a sink point while the IMO or object appears stationary 
or slower than the background. Otherwise, the overall cue is 
given as sink point (8 < 0). The unit for this cue is 1/s or Hz. 
This expansion/contraction cue has been studied and discussed in 
the psychophysical literature (Koenderink and van Doorn, 1976; 
Braddick, 1993). 

We define accelerations/decelerations as a third cue. These are 
2nd order temporal derivatives of points projected onto the image 
plane much like the acceleration and deceleration known from 



classical mechanics (Goldstein et al., 2001). Considering such 
derivatives are motivated by studies demonstrating sensitivity to 
such cues (Angelaki et al., 1993; Cao et al., 2004). Similarly to 
the accretion and deletion cue, we define acceleration and decel- 
eration along the normal of a local edge that is defined by the 
angle p. A combination of these desired properties leads to the 
definition: 




This cue indicates acceleration for a > 0 and it indicates deceler- 
ation for a < 0. The units of this cue are m/s^. 

The fourth cue is defined as local, spatial curvature. We stress 
the fact that this is a local cue, since we use a Taylor series approx- 
imation up to the 2nd order to derive it. Thus, the local curve in 
the image plane is defined by: 

We use this local approximation since the analytical solution 
[x(t), y(t)] for the non-linear, coupled system of two ordinary 
differential equations (ODEs) as defined by Equation (1) might 
be unknown. Assuming our local approximation, the curvature 
is computed by det(X, X)/ . Plugging in the definition of 
Equation (22) yields: 

1} = 1}; - iJb with 

XB/l Xb/i ■ (dxXB/l) + h/I ■ (9yXB/l) 

h/i XB/I ■ (dxh/i) + h/i ■ (dyh/l) , , 

%i = 3 • (23) 

The Equation (23) defines the curvature difference between IMO 
and background. The units of this curvature are 1/m. This cur- 
vature can be interpreted as the curvature of integral curves 
that are solutions to the system of ODEs in Equation (1). 
This curve has a concave shape for 1} > 0 and a convex shape 
for < 0. 

As a fifth cue for segmentation we define local, temporal cur- 
vature. In contrast to the local, spatial curvature that is defined in 
the image plane, this curvature is defined for a single location in 
the visual field along the temporal domain. This assumes that the 
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curve is defined as {x(t),y{t)) for one fixed image location. Thus, 
the curvature is: 



Yi-YB 



with 



YB/I 



xb/i ■ yB/i — yB/i ■ xb/i 



(24) 



^B/I 



+yl/i 



This curve has a concave shape for y > 0 and a convex shape for 
Y < 0. The units for this curvature are 1/m. Table 4 summarizes 
the definition for these five segmentation cues. 

We evaluate the segmentation cues for the scene and motion 
configurations as given in Warren and Saunders (1995), Royden 
and Hildreth (1996), and Fajen and Kim (2002) and summarize 
the analytical expressions for all cases and experimental config- 
urations in Table 5. In the case of linear motion curvature cues 
{■& or y) are absent, and in the case of curvilinear motion above 
a ground-plane cues of acceleration/deceleration and local, tem- 
poral curvature (a and y) are absent; Table 5 shows that the cue 
values are set to zero in these cases. 

We further evaluate these segmentation cues from Table 5 for 
their contribution to an overall segmentation of an IMO from 
the background. Figure 2D shows the integration along the con- 
tour of an IMO while considering the normal vector of the local 
edge. A combination of values of each of the four line segments is 
achieved by summing their absolute value. We take the absolute 
value for the following reason: assume an object has a leftward 
motion and appears in front of a rightward moving background. 
This gives an accretion on the right edge of the IMO and deletion 
on the left edge of the IMO. The sum results in a zero-net seg- 
mentation. To avoid such a zero-sum result, we take the absolute 



Table 4 | Defines analytical models for five segmentation cues of 
independently moving objects or stationary objects In the scene. 


Cue 


Model term 


Accretion (A > 0) and 
deletion (A < 0) 


A = Ae-A,withAe,= (^«/')7'^°^f) 
XYb/i / \sin(P) / 


Expansion (5 > 0) and 
contraction (8 < 0) 


5 = 5/ - Sg with 5g/; = dxXB/l + dyi/B/l 


Acceleration (a > 0) and 
deceleration (a < 0) 


/xg/,\7cos(fi)\ 
a = a; - ag with ag/, = 1 . ' 

\ Yb/i / \ sin(P) / 



Local, spatial curvature: 
concave (iJ > 0) and 
convex {a < 0) 



1 = i}/ — i}g with 

><B/i 'xb/i ■ (Sx><B/i) + Yb/i ■ (dy><B/i) 
Yb/i Xb/i ■ (SxYb/i) + Yb/i ■ (SyYB/i) 



"B/i -- 



Local, temporal curvature: y = y/ — yg with yb/i 
concave (y > 0) and 
convex (y < 0) 



B/I + ^B/l 

_ Xb/i ■ Yb/i - Yb/i ■ xb/i 
'J^B/i + yi/i 



The sign of these defined expressions has the interpretation as indicated in 
the left column. The local, spatial curvature is approximated by a Taylor series 
expansion up to the 2nd order 



value for contributions of each line segment. Due to taking the 
absolute value, the interpretation of the sign that we provided in 
Table 5 is no longer valid since all values are positive. We evalu- 
ate the segmentation cue along the contour using the following 
integrals: 



ryi + h[/2 

/ /v(x; + w;/2,y;P = 0°) 

Jy, - h[/2 



nx[ - wi/1 

+ / fv{x,yi + V2; P = 90°) (fcc 



1x1 + will 
(•yi - hl/2 

hi + hi 12 
fXi + wi/2 



ryi - hi/2 

+ / fvixi - wi/2,y;P = 180°) dy 

Jvi + hi/2 



+ / fv{x,yi - hi/2; P = 270°) dx\ . (25) 

J X[ — W[/2 I 

In Equation (25) we use the generic function /„ that can be any 
of the cues v e { A, 8, a, -Q, y) defined in Table 5. Note that this 
function depends on the image location (x, y) and in cases of 
vector-valued cues also on the orientation of the edge normal p. 
For the definition of the combined segmentation cue, we use a 
linear superposition of the segmentation cues accretion/deletion, 
expansion/contraction, and acceleration/deceleration. Thus, the 
combined segmentation cue is: 



5 = Ca • Sa + C5 ■ S5 + Ca ■ Sa 



(26) 



with coefficients ca, cs, and that are altered to fit the data. 
We incorporate this segmentation cue into methods estimat- 
ing the horizontal FOE position or path radius by weighting 
integrals ranging over background and IMO regions in the 
image plane accordingly. IMO regions are weighted by (1-s) 
where 5 ranges between zero and one. For a full segmenta- 
tion, s = 1, this discounts the region where the IMO is present 
for the estimation of the background motion. The Equation 
(26) does not include available curvature cues, since analyti- 
cal expressions for the corresponding line integrals are difficult 
to find. 

RESULTS 

We organized the results into four subsections, beginning with a 
description of heading bias and path perception for estimation 
methods which do not use segmentation. 

LINEAR LEAST SQUARE ESTIMATION OF HEADING OR PATH (RADIUS) 
CANNOT EXPLAIN OBSERVED BEHAVIOR 

We study the validity of analytically derived bias terms for trans- 
lational and curvilinear path motion by comparison to related 
psychophysical studies (Warren and Saunders, 1995; Royden and 
Hildreth, 1996; Fajen and Kim, 2002). 

In the first set of experiments, following Warren and Saunders 
(1995), the scene is defined by two fronto-paraUel planes, one for 
the background and one for the IMO, both at an initial (f = 0) 
distance of dg o = dj^ = 10 m. Their speeds along the optical 
axis are v^ b = 2 m/s and v^j = 3 m/s, respectively. The heading 
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direction of the background appears in the horizontal at any 
combination of probe locations at ±10°, ±8°, ±6°, or ±4° and a 
variable heading around that probe of ±4°, ±2°, ±1°, or ±0.5°. 
This gives 16 combinations at which background heading can 
appear. In addition, the IMO appears at a horizontal angle ±6° 
at the opposite side of the probe. The path angle is defined as the 
angular difference between the motion direction of background 
and the motion direction of the IMO. These experiments probe 
horizontal path angles of —6°, 0°, and +6°. All these parame- 
ters assume that motions are defined with respect to a stationary 
camera. Our model assumes an instantaneous flow field (snap- 
shot) as it is present at the end of each stimulus presentation. For 
simplicity, we excluded a temporal integration in our analytical 
models. The stimulus is present for 1 .5 s. Thus, the size of the IMO 
at the end of the stimulus presentation has increased by a factor of 
1.6 due to looming. At the end of each stimulus presentation the 
IMO has a size of 1 6° x 16°. For the model we use the display size 
of 32° X 32° or = 1 cm and hs = I cm with a focal length of 



/ = 1.74 cm. The original study used a display size of 40° x 32°. 
We simplified the model by using a square display. 

Figures 3A-D shows the data and model results side by side. 
All four graphs report the heading error over path angle. The 
heading error in the model is denoted by r\E and the path angle 
by TiB — Tii. The reporting of heading bias is divided into two 
groups. In one group the FOE of the background is visible and 
in the other group, the FOE of the background is obscured by the 
IMO. In the visible case, the heading error for humans is small 
with a constant bias underestimating the heading. This means 
that the FOE of the background is estimated to be closer to the 
center of the screen, see Figure 3A. Open circles in the graph of 
Figure 3A depict the error for an opaque IMO, graphs with closed 
circles encode the error for a transparent IMO. This underesti- 
mation of heading is a known effect reported in studies without 
IMOs (Warren and Kurtz, 1992; Crowell and Banks, 1993). In the 
model, heading errors appear large although cases of zero error 
appear within the standard deviation, regardless of the path angle. 
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FIGURE 3 I Displays heading biases and perceived path from 
experimental data (Warren and Saunders, 1995; Royden and 
Hildreth, 1996; Fajen and Kim, 2002) and our analytical model 
without segmentation cues cannot explain the data completely. 
(A,B) Sliow tine Ineading error for humans and our model, respectively, 
when the FOE is visible. (C,D) Show the heading error for humans 
and our model, respectively, when the FOE is covered by the IMO. 



(E,F) Show the estimated heading for humans and our model, 
respectively, when the IMO's path and the observer's path are not 
crossing. (G,H) Show the estimated heading for humans and our 
model, respectively, for crossing paths. (I) Percent correct responses of 
humans estimating future path for four different configurations labeled 
as A, B, C, and D. (J) Error of the radius estimated by our model for 
the same configurations. 
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see Figure 3B. For the case of the obscured FOE humans have a 
bias in heading perception, see Figure 3C. In the black condition 
(squared symbols) the IMO region is blanked out such that no 
texture appears in this region. The model replicates this behavior 
except for the shift in estimating heading closer to the center of 
the screen, which is not an effect of the IMO, see Figure 3D. 

The second set of experiments follows the study of Royden and 
Hildreth (1996). In their experimental design background and 
IMO appear as fronto-paraUel planes. The background is initially 
at the distance do s = 10 m and the IMO stays constantly at the 
distance do,/ = 5 m, moving along with the observer. However, 
the IMO changes its horizontal distance from the observer by 
moving sideways with a velocity of 8. l°/s (0.14 rad/s) in the image 
plane. For the frame rate of 25 Hz, as used in the experiment, 
this translates into a horizontal velocity of v^j = 5m-tan(0.14 
rad/s/25 Hz)-25 Hz or v^j = 0.71 m/s. Assuming a stationary 
observer, the background moves by Vz^b = 2 m/s. In the exper- 
iment the FOE for the background appears at —2°, 0°, or +2° 
vertically and 4°, 5°, 6°, or 7° horizontally. The field of view is 
30° X 30°. In our model we choose a display size of 1 cm x 1 cm 
and adjusted the focal length for a 30° visual field, / = 1.87 cm. 
The experiment is composed of two conditions. In the non- 
crossing condition the IMO's path does not cross the observer's 
path and in the crossing condition it does. For the non-crossing 
condition the IMO starts at the horizontal position xj = 0.6° and 
for the crossing condition it starts at x; = 10.7°. In all trials, the 
projected size of the IMO stays constant with 10° x 10°. 

Figures 3E-H shows the reported heading of the observers in 
the presence of an IMO from psychophysical experiments and our 
analytical model. In the non-crossing condition heading errors 
for humans are small; see Figure 3E. Our analytical model instead 
shows large heading errors; see Figure 3F. Note the difference in 
axis scaling between Figures 3E,F. In the crossing condition the 
perceptually reported heading error is large and tends to under- 
estimate the true heading, dashed line in Figure 3G. Again the 
model's error is larger, see Figure 3H. 

The third set of experiments studies path perception, following 
the study of Fajen and Kim (2002). The observer and IMO orbit. 
From the viewpoint of the observer or IMO, this is a curvilinear 
motion. In all conditions the background (or observer) moves 
with the constant, linear velocity vg = 13.2 m/s, and the rota- 
tional velocity changes according to the radii rg = 80 m, 120 m, 
160 m, and 320 m. For these changing radii the rotational veloc- 
ity is M = v/r; or m = 9.5°/s, 6.3°/s, 4.7°/s, and 2.4°/s. The 
rotational velocity of the IMO is 2.86°/s slower than that of the 
background. Thus, the object does not approach the observer as 
fast as the background does. It moves from near to far while the 
background moves from far to near. We test four different IMO 
paths: the IMO starts and remains to the right (path A) or to the 
left (path B) of the observer, or the IMO's radius decreases after 
initially appearing to the right (path C) or to the left (path D) 
of the observer. In the last two conditions, the IMO's path and 
the observer's paths are crossing, and 2 s after the start, the radius 
of the IMO either increased or decreased by 7.2 m. Thus, r; = 
vb ± 7.2 m. For the evaluation of the analytical model we assume 
an eye height ofh = —1.65 m and a gaze parallel to the ground. 
Furthermore, we assume the IMO appears at yj = —5° below 



the horizon and is represented as a fronto-paraUel plane 5 m 
away from the observer, covering 5° x 5° of the observer's visual 
field. Fajen and Kim (2002) did not describe these IMO parame- 
ters for their study. We assume a square display of 32° x 32° to 
simplify the analytical model instead of 42° x 32° used in the 
experiment. The procedure of the experiment included humans 
indicating their future path by reporting whether they would pass 
a presented pole left or right. The accuracy of path perception was 
reported as a percentage of correct responses percentage over all 
trials. 

Figures 3IJ show results of curvilinear path perception 
for humans and our analytical model. Between conditions 
(path A-D), the correct detection rate is not significantly dif- 
ferent, see Figure 31. However, trajectories with high curvature 
(small r) are more difficult to predict than those of low curva- 
ture (large r). For the analytical model we report the ratio pr of 
estimated radius te to ground-truth radius rg of the background 
motion: Pr = rg/rB- We note that the future path can be pre- 
dicted solely based on the radius of the circular motion, assuming 
that this radius stays constant with time and that the optical axis 
is tangent to the trajectory; see Equation (16) in the Methods. 
Therefore, we assume the error measure is informative about the 
paths chosen by humans. We depict the error for our analytical 
model in Figure 3J. Errors above one indicate an overestimation, 
errors below one an underestimation, and values close to one rep- 
resent a small error. Paths A and B have the same error and only 
one curve is plotted. Note that the trend for the model is the same 
as in the data, compare Figures 3IJ. High curvatures result in 
large errors or a low correct detection rate. Low curvatures have 
small errors or a high correct detection rate. 

Thus far, our analytical model does not correctly explain all 
psychophysical data. One shortcoming of the model is the inte- 
gration of motion across the entire image including IMOs; how- 
ever, IMOs can be segmented and, thus, effectively discounted 
before estimating heading or path (Adiv, 1985). We foUow this 
idea and study five segmentation cues for IMOs, most of which 
also account for the case of stationary objects in front of the 
background. 

SEGMENTATION CUES ARE STRONG AT DEPTH DISCONTINUITIES 
INTRODUCED BY OBJECTS AND IN MOST CASES ARE ALTERED BY 
MORE THAN 30% IF THESE OBJECTS MOVE INDEPENDENTLY FROM 
THE BACKGROUND 

Our aim is to show that segmentation cues for IMOs are different 
from those at depth discontinuities in most cases, which should 
allow the observer to distinguish between IMOs and depth dis- 
continuities. We study each of the segmentation cues for ( 1 ) only 
a depth discontinuity or a static discontinuity and (2) a depth and 
motion discontinuity or dynamic discontinuity (IMO). 

Segmentation cues for linear motion toward fronto-parallel planes 

The depth discontinuity is defined by assigning the distance do to 
the background and the distance do — Ad to the object. For trans- 
lational motion toward fronto-paraUel planes, we assume that the 
object motion is defined by adding ( Avv-, Avy, Av^) to the back- 
ground motion (v^, Vy, Vz). A dynamic discontinuity introduces 
the foUowing accretion/deletion: 
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Ad ^ ^ 1 ^ ^ , ^ 

■— — — -a{vB) + —a(Av) «p with 

(do - Ad) -do do ' 



-f-Vy+y-Vz 



and 



= (cosp, sinP). 



(27) 



Note that the 2nd term in A of Equation (27) is introduced 
by the IMO and, thus, can increase or decrease this cue, which 
depends on the signs of the two terms in A. In the case that 
fl(vB) and 3(Av) fall into different half-planes defined by the 
normal n and Ad < do, then the cue is enhanced. For most 
situations Ad < do holds since the relevant object appears in 
front of the observer as opposed to Ad > do, where it appears 
behind the observer. In general, it is important that the cue 
difference between static and dynamic depth discontinuity is 
strong. To give a qualitative answer we evaluated this dif- 
ference for p = 0°, X, y e {— 5, — 2.5, 0, 2.5, 5}mm, v^b, Vysd 
{-15, -10, -5, 0, 5, 10, 15}°, Avv, Av^t {-5, -3, -1, 1, 3, 5}°, 
doe {5, 10, 15, 20, 25, 30} m. We combine this distance do with 
Ad being 0.5, 0.75, or 0.875 of do- All other parameters are 
set as in the experiment of Warren and Saunders (1995). We 
compute A for a static depth discontinuity choosing the back- 
ground motion A(Av = 0) and count a difference if | A(Av = 
0) - A(Av)| is above 30% of |A(Av = 0)|, the condition for 
a static depth discontinuity. This criterion is largely based on 
speed differences between object and background motion which 
was attributed importance in the segmentation of IMOs (Royden 
and Moore, 2012). Following this criteria, 63% of all evalu- 
ated cases show a 30% difference between static and dynamic 
discontinuity (IMO). 

For expansion/contraction cues we use the same definitions 
for the distance of IMO and background, and their motions. This 
results in the cue: 



do — Ad \ do 



Ad 



+ Av: 



(28) 



The IMO motion introduces the 2nd term in Equation (28). 
Most times humans go forward (Vz > 0). It holds that Av^ > 0, 
if the IMO approaches the observer faster than the background, 
which is the case in the Warren and Saunders (1995) experi- 
ment. Our quantitative analysis gives a 30% difference between 
a static and dynamic discontinuity (IMO) in 83% of all tested 
cases. 

The acceleration and deceleration cue for the same scene and 
motion configuration is given by: 



2do - Ad Ad 



Ad)2 dl 



Vz,b3(vb) + 



AVy, 



[do - Ad)2 



fl(Av) (29) 



where a(y) and are defined as in Equation (27). The dif- 
ference in motion Av between background and IMO intro- 
duces the 2nd term in Equation (28). Whether this motion 
difference introduces an increase or decrease depends on the 
signs of each term. Since there are many cases, we restrict the 



description to a qualitative analysis with the previously described 
parameters. Eighty-three percent of all evaluated cases show 
a thirty percentage difference between a static and dynamic 
discontinuity (IMO). 

The curvature cues for the translational motion are zero- 
valued in all cases, regardless of the scene geometry, see Table 5. 

Segmentation cues for curvilinear motion above a ground plane 

As next configuration, we study curvilinear motion defined by 
the linear velocity v = M-r and the yaw-rotational velocity oo. 
The scene is defined as one fronto-parallel plane at the distance 
dj, which is the IMO, and one ground-plane, the background. 
The depth discontinuity is expressed in the same way as before; 
however, the motion for the object oo -|- Am and r + Ar is 
now defined using the components Ar and Am. For a depth 
discontinuity the accretion/deletion cue evaluates to: 



ffh — diy _ o) ■ Ar-I- Ato ■ r-l- Ato ■ Ar^ Aco- , ^ 



with 3c' = (x,y), b' = (f^ + , xy), 
and jip = (cosp, sinP). 



(30) 



The motion of the IMO introduces the 2nd and 3rd term for A. 
Due to the various signed terms involved, we chose a quantitative 
analysis rather than denoting all cases separately. For our quan- 
titative analysis we vary parameters in the following ranges: Ar 
€ {-10, -8, -5, -1, 1, 5, 8, 10} m. Am 6 {-0.2, -0.15, -0.1, 
-0.05, 0.05, 1, 0.15, 0.2}rad/s, v € {-15, -14, -13, -12, -10, 
10, 12, 13, 14, 15} m/s, r € {80, 120, 160, 320} m. All other param- 
eters are kept the same as in the experiment of Fajen and Kim 
(2002). For this setting 91% of all evaluated cases provide a differ- 
ence above 30% between static and dynamic discontinuity (IMO) 
taking the static discontinuity as reference. 

For the next cue we analyze expansion/contraction for the 
same configuration. This cue evaluates to: 

Ufh - 3diy 



difh 

M • Ar-F Am • r-l- Am • Ar 3x 

+ 2 -Am. 

di f 



(31) 



Note that the motion discontinuity adds the terms Ar and Am. 
The additional rotational velocity Am introduces a dependency 
on the horizontal position in the image plane. Our quantitative 
analysis yielded a 30% difference between static and dynamic 
discontinuity in 89% of all cases. 

Next, we derive the acceleration/deceleration cue for curvilin- 
ear motion keeping the configuration of scene and motion the 
same as before. This cue evaluates to: 



xdi + (r+ Ar)/ 



(m -I- Am)^ (r -I- Ar) • x^Mp. 



(32) 



Note, that the ground-plane has a zero-valued temporal deriva- 
tive. Thus, a strong difference occurs due to the varying 
motion for the condition with and without IMO. This strong 
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difference is also reflected in a 98% occurrence of a 30% 
difference. 

The cues for local spatial curvature and local temporal curva- 
ture are summarized without giving the explicit equations. For 
local, spatial curvature cues a 30% difference appears in 20% 
of all cases and for local, temporal curvature cues such differ- 
ence occurs in 21% of all cases. Due to the absence of temporal 
and spatial curvature cues in the translational motion case and 
no strong differences for spatial and temporal curvature between 
static and dynamic objects (IMOs), we excluded these two cues 
from the further analysis. 

CONTOUR SEGMENTATION CUES FOR IMOs ARE DEFINED ALONG THE 
TRANSITION EDGE BETWEEN IMO AND BACKGROUND 

Figure 4 shows the evaluation of these contour segmentation 
cues for different sized IMOs. We kept parameters the same as 
described before and additional parameters are set as follows: p = 
45°, Vx,B = 10°, Vy^B = 0°, v^j = 2°, and Vyj = 0°. Figure 4A 
shows one exemplar configuration from the Warren and Saunders 
(1995) set of experiments, and Figure 4B shows one exem- 
plar configuration from the Royden and Hildreth (1996) set of 



experiments for: v^^b = 5° , Vy^B = 0° , and P = 45° . Figures 4C,D 
show accretion/deletion cue strengths for these two examples 
when varying the horizontal position of the IMO and its size. 
In both cases the trend of the data from the simulation is the 
same with strong cues being present on the right-hand side of 
the visual field where the FOE of the background is located. This 
is explained by the definition of our measure for contour seg- 
mentation. Accretion/deletion is measured normal to the edge, 
and we compute the sum of absolute values taken for each line 
integral [compare with Equation (25) and Figure ID]. Cues for 
the upper and lower edge of the IMO are strongest if this IMO 
is located close to the FOE of the background, which explains 
the trend given in the data which graphs of Figures 4C,D show. 
These cues increase with the size of the IMO as well. The accre- 
tion/deletion cue in the Royden and Hildreth (1996) example 
is stronger than in the Warren and Saunders (1995) example. 
Figures 4E,F show expansion and contraction cues. These cues 
are not only independent of the horizontal or vertical position 
of the FOE but also independent of the spatial location of the 
IMO, see Table 5. Figures 4G,H show acceleration/deceleration 
for the two experiments. For this case the trend in the curves is 



Warren & Saunders 



IMO 



-6° 
+ 

X, 



background 

i+2° +10° 
FOE, FOE„ 
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• + 
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FIGURE 4 I Shows graphs for the evaluated line integrals of 
segmentation cue strengths along the contour of the IMO. Due to 

tine difference in the graph's shape depending on the IMOs horizontal 
position, these cues form a "basis set" to selectively down-regulate the 
heading bias or miss-perception in the path for the model. (A) One 



example of the Warren and Saunders (1995) experiments. (B) One 
example of the Royden and Hildreth (1996) experiments. (CD) show the 
response for accretion and deletion, (E,F) for expansion and contraction, 
and (G,H) for acceleration and deceleration for the two examples from 
(A,B), respectively. 
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different between the two examples. In Figure 4G the strength 
of the cue increases the further the IMO is shifted toward the 
right side of the visual field, toward the FOE of the background 
motion. When IMO and FOE overlap, vectors of the IMO and 
the background point approximately in the same direction while 
their lengths differ. Lengths of vectors that belong to the IMO 
are longer than those from the background. The length of vec- 
tors from the IMO scales by l/d^. In comparison, the length of 
vectors from the background scales by 1/do. For the Royden and 
Hildreth (1996) example, which Figure 4H shows, the trend is a 
U-shaped curve. When the IMO overlaps with the position of the 
background FOE the acceleration/deceleration is weakest. In this 
example, all flow vectors of the IMO point to the left. Thus, if the 
IMO is located on the right-hand side of the visual field then this 
cue has a small magnitude, since background vectors and IMO 
vectors point largely into the same direction. In contrast, if the 
IMO appears on the left-hand side of the visual field, vectors of 
IMO and background point into opposite directions, leading to a 
strong acceleration/deceleration magnitudes. 

To fit human heading biases, which are large for a crossing 
IMO, in the model we mainly use acceleration/deceleration as 
a segmentation cue, since it has the appropriate characteristics 
for reducing the heading bias in the correct conditions: accel- 
eration/deceleration is strong when the IMO's position is far 
from the background's FOE, which is the non-crossing case. In 
this case the cue can effectively reduce the bias term. When the 
IMO's position is close to the background's FOE the accelera- 
tion/deceleration is weak and, thus, the bias largely remains. The 
next subsection will discuss this idea in more detail and provides 
data fits. 

INCLUDING SEGMENTATION CUES INTO A LINEAR LEAST SQUARE 
ESTIMATION EXPLAINS MEASURED HEADING BIASES AND 
CURVILINEAR PATH PERCEPTION 

To fit the data on heading biases and path perception, we 
use accretion/deletion, expansion/contraction, and accelera- 
tion/deceleration cues. These three cues have different graphs 
depending on the horizontal position of the IMO. Some graphs 
have zero slope, others have a left/rightward slope, and yet others 
have multiple slopes. This difference in characteristics enables us 
to use them as a "basis set" of components to selectively down- 
modulate heading biases. Our aim was not to find the best fit, 
e.g., lowest r- -error; rather we fit the curves by hand, changing 
the weights of these three segmentation cues using a weighted 
linear combination (see section Methods). Figure 5 shows the 
graphs for heading bias and path perception using existing psy- 
chophysical data for the fit (Warren and Saunders, 1995; Royden 
and Hildreth, 1996; Fajen and Kim, 2002). Parameters for the 
experiment settings were kept the same as described for Figure 3. 
We show results from Figure 3 to allow for a better compar- 
ison between the data, our model without segmentation cues, 
and our model with segmentation cues. Figures 5A-F shows 
the heading bias for the conditions of visible (first row) and 
obscured FOE (second row). The data fit for Figures 5A-F has the 
parameters: ca = 10^, = —6 x 10~, and Cq, = 4.5 x 10^ [com- 
pare with Equation (26)]. Including the segmentation cue into 
our model allows us to account for the perceptual observations 



and, thus, the fitting of behavioral data: for the condition of 
a visible FOE, the mean values for heading biases range all 
around zero while providing a flat graph that fits better to the 
characteristics of data, as seen in Figure 5C. Without segmen- 
tation the graph showing the simulation results has a positive 
slope, see Figure 5B. For the obscured FOE, including the seg- 
mentation cue into the model does not change the heading 
biases substantially (compare Figures 5E,F). However, the stan- 
dard deviations are slightly reduced for the model with seg- 
mentation compared to the model without segmentation, which 
would make a significance test against a zero bias stronger. Note 
that both models reflect the data plotted in Figure 5D. The 
segmentation cue could not generate the bias of estimated head- 
ing toward the center of the screen as visible in the data, see 
Figures 5A,D. 

Figures 5G-L show results for the next set of experiments, 
the 3rd row for a non-crossing IMO and the 4th row for a 
crossing IMO. Parameters of the fit are: ca = 3.5 x 10^, Cf, = 
— 10.5 X 10^,andca = 3 x 10^. In this case the segmentation cue 
helps to improve the results and reduce the deviation between 
ground-truth horizontal heading (diagonal in the graphs) and 
estimated horizontal heading. Data from Royden and Hildreth 
(1996) is plotted in Figures 5G,J for non-crossing and cross- 
ing IMO, respectively. Model results without segmentation pro- 
duced large deviations as visible by comparison of the graphs in 
Figures 5GJ with those in Figures 5H,K. Note that the panels use 
different scales. Including the segmentation cue into the model 
helped to reduce this deviation, in particular in the non-crossing 
IMO case, see Figure 51. For the crossing case the model with seg- 
mentation overestimates the heading by about 2° independent of 
the heading; this is similar to the data in terms of the strength 
of deviation. Data from the human subject shown in Figure 5J 
underestimated heading by about 2° . The difference in bias direc- 
tion is not critical as Royden and Hildreth (1996) point out that 
the direction of deviation was not consistent across subjects. 

Figures 5M-0 shows the estimation of curvilinear path for 
data, the model without segmentation, and the model with seg- 
mentation, respectively. Parameters of the fit are: ca = 10*, = 
lO', and Coi = 10*. The first two panels were described along 
with Figure 3. Including the segmentation cues shifts the data 
points for the radius r = 320 m closer to a unit ratio and, further- 
more, reduces the deviation for smaller radii; see Figure 50. Note 
that the trend, smaller error for high curvatures, is maintained 
when incorporating the segmentation cue into the model. The 
important observation is that various cues of segmentation can 
be viewed as forming a "basis set" of the necessary constituencies 
to be linearly combined in order to selectively reduce deviations 
for configurations in which either the IMO's path is crossing the 
observer's path or the IMO is obscuring the position of the FOE 
from the background motion. 

DISCUSSION 

We derived expressions for the heading bias and radius esti- 
mate for linear motion and curvilinear motion as well as five 
segmentation cues. These cues are accretion/deletion, expan- 
sion/contraction, acceleration/deceleration, spatial, and temporal 
curvature. Such cues and their definition are general enough to 
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FIGURE 5 I Shows data from the original experiments (1st column) 
and our model without (2nd column) and with segmentation (3rd 
column). Wliile tine model without segmentation did not fit the data very 
well, the model with segmentation does so for all three sets of 
experiments (Warren and Saunders, 1995; Royden and Hildreth, 1996; 
Fajen and Kim, 2002). (A,B,C) Show the heading error for humans, our 
model without and with segmentation, respectively, when the FOE is 
visible. (D,E,F) Show the heading error for the same three categories for 



an obscured FOE. (G,H,I) Show estimated heading for humans, our model 
without and with segmentation, respectively, when the IMO's path and 
observer's path are not crossing. (J,K,L) Show the estimated heading for 
the same three categories for crossing paths. (M) Percent correct 
responses of humans estimating future path for four different 
configurations labeled as A, B, C, and D. (N,0) Error of the radius 
estimated by our model for the same configurations without and with 
segmentation, respectively. 



be studied for real-world video, e.g., in combination with exist- 
ing bio-inspired flow detection models (Pauwels et al., 2010), see 
also Figures IC-G. A linear method of flow-based estimation for 
heading or path radius could not explain the observed human 



biases. Humans show heading biases if the IMO covers the FOE 
of the background motion or the IMO's path intersects with 
the observer's path (Warren and Saunders, 1995; Royden and 
Hildreth, 1996). However, humans accurately predict future path 
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for curvilinear motion in the presence of IMOs. These seem- 
ingly contradictory results might be explained by segmentation 
cues utilized by the visual system to improve the judgments 
about heading or path as these cues differ between experimental 
conditions. In studies that probed heading, fewer segmentation 
cues are available in theory than in those studies that probed 
future path. In particular, local spatial and temporal curvature 
cues are absent for linear motion used to probe heading. Our 
linear method for the estimation of heading or path radius 
improved when integrating segmentation cues. This improve- 
ment is expressed in a better fit to data from psychophysical 
experiments. We use different weights for the segmentation sig- 
nals in each study, which we do not view as critical since motion 
and scene parameters differed widely between these studies. Our 
computational study suggests that segmentation cues play a role 
in the perception of heading and path when IMOs are present. 
Those cues are used to reduce or discard the IMO's influence 
for motion integration. We discuss this suggestion in the con- 
text of recent studies on heading and path perception that present 
alternative views. 

CONCEPTUAL AND COMPUTATIONAL MODELS FOR THE 
SEGMENTATION OF IMOs AND ESTIMATION OF SELF-MOTION 

An alternative suggestion to our flow segmentation is flow pars- 
ing, the segmentation of IMOs employing global cues rather 
than a local flow segmentation that we proposed. Warren and 
Rushton (2009a) provide evidence that humans use such flow 
parsing strategy in which the majority of the flow is consistent 
with self-motion through a hallway. The IMO appears as a cir- 
cular disk falling to the ground. The proposal is that humans 
employ a three-step procedure to recover all motions in the stim- 
ulus. First, the visual system accesses the self-motion based on 
dominating flow patterns from the background. Second, the glob- 
ally predicted flow for this self-motion is subtracted from the 
sensed flow. Third, the residual motion is the motion of the IMO 
that can be estimated. Aside from semantic cues (falling ball) 
being present and choosing a parameterization where no heading 
bias was reported before (compare Warren and Saunders, 1995; 
Royden and Hildreth, 1996), Warren and Rushton do not explain 
how the scene is initially segmented to access self-motion from 
flow originating from the background. In a recent study Warren 
et al. (2012) point out that flow parsing might not depend on 
prior heading estimation. In contrast, our study provides a spec- 
ification of the mechanisms for such an initial segmentation that 
could be used to estimate self-motion from the background flow 
mainly. The features and properties are defined on the basis of 
monocular depth cues that could be enhanced even further using 
available binocular depth cues (Warren and Rushton, 2009b). 

An alternative hypothesis suggests combining segmentation 
and self-motion estimation. Hildreth (1992) and Royden (2002) 
proposed a computational model that employs motion-opponent 
operators. These operators locally compute the motion differ- 
ence and only strong differences are further considered. A voting 
strategy with these motion differences yields the translational 
self-motion velocity assuming that the background subsumes the 
largest area in the image. The motion-opponent computation 
segments linear from rotational velocity (Rieger and Lawton, 



1985) and the voting mechanism segments self-motion from 
IMOs. Due to voting biases occur in Royden's model when back- 
ground and IMO motion are similar, e.g., when their paths cross. 
Our model explains the data (Royden and Hildreth, 1996) and 
extends it by providing an explanation for curvilinear path data 
(Kim, 2000; Fajen and Kim, 2002). 

Adiv (1985) suggests a similar computational approach that 
uses the optic flow Equation (1) for plane models. Assuming 
that the scene can be approximated by local planar patches, all 
parameters — linear motion, rotational motion, and those of the 
plane — are constant within such a local region. A voting (group- 
ing) mechanism for such parameters that are locally constant can 
indicate an IMO who wiU disagree due its motion difference. 

Layton et al. (2012) suggest a partly recurrent neural network 
of motion integration mechanisms that fits data of heading biases 
for linear path perception; however, it remains unclear if the same 
model mechanisms could account for the data of curvilinear path 
perception. 

Pauwels and Van HuUe (2004) propose an iterative mecha- 
nism to combine segmentation and estimation of self-motion. 
The first step estimates the self-motion parameters by employ- 
ing a weighted, adaptively de-biased bilinear constraint (Bruss 
and Horn, 1983). The second step computes a residual transla- 
tional flow by subtracting the estimated analytical rotational flow 
from the input flow. This residual translational flow is compared 
to the analytical translational flow based on the angular differ- 
ence between individual flow vectors. This difference is large for 
regions that contain IMOs and small for the background, and is 
used to define weights. These weights are the "glue" between first 
and second step of the iterative optimization process and segment 
IMOs from the background. 

Other methods solve segmentation and heading estima- 
tion with the expectation and maximization (EM) algorithm 
(MacLean et al, 1994; Clauss et al., 2005). The expectation step 
estimates the probability of a data point as being generated by 
a motion model. Maximization is achieved by formulating a 
probability weighted least squares problem for the estimation of 
self-motion or object-motion from optic flow. Both steps are iter- 
ated until convergence is reached. In practice, this convergence 
is assumed when the change in probability assignments between 
successive iterations is below a threshold. 

Another strategy to solve the intertwined problem of segmen- 
tation and motion estimation uses the random sample consensus 
(RANSAC) method (Fischler and BoUes, 1981). This method iter- 
atively finds a set of data points, which is consistent with the 
self-motion model assuming that the background subsumes the 
largest area in the image (Raudies and Neumann, 2012). All these 
approaches do not require a prior segmentation as suggested in 
our work; however, they could profit by it. Segmentation could be 
included as a prior for the voting methods and as an initialization 
for the iterative methods. 

STEERING CONTROL 

Our analytical models have implications for steering control. 
Cutting et al. (1995) suggest that stationary obstacles are avoided 
by using motion parallax while keeping fixation on the object 
off to one side. We used similar motion models to qualitatively 
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evaluate the scenario of avoiding a stationary obstacle (Raudies 
et al., 2012). Here, we further refined these models to account 
for IMOs, which shows that segmentation cues of moving obsta- 
cles are enhanced in most cases compared to stationary ones. 
Fajen and Warren provided data and a dynamical force model 
for the avoidance of stationary/moving obstacles and the pur- 
suit of a moving target (Fajen and Warren, 2003, 2004; Warren 
and Fajen, 2008). Their model uses a birds-eye view perspective, 
assuming knowledge about the position of self and obstacles in 
polar coordinates. Goals act as attractors and obstacles as repellers 
in a 2nd-order dynamical system. 

Other models take a first-person perspective approach for 
steering control (Browning et al., 2009; Elder et al., 2009). These 
models use analytical flow or detected optic flow to estimate 
the horizontal position of objects and heading. The estimated 
object positions are fed into a control circuit where obstacles 
are represented by negative Gaussians attached to their horizon- 
tal position in the visual field, the target by a positive Gaussian 
represented at its horizontal position, and the horizontal head- 
ing as another positive Gaussian. Superimposed entries from all 
horizontal positions are summed and the resulting distribution 
defines the steering variable for the yaw velocity. 

Our study differs by using an analytical approach. The derived 
segmentation cues extend the overall functionality of models that 
have been previously proposed. In addition to deriving these cues, 
we incorporate them into a mechanism that selectively integrates 
motion in order to estimate heading or path. This first-person 
perspective also accounts for the size of target or goal object in 
the visual field. Such an approach could implicitly account for the 
distance dependence in the dynamics due to the effect of looming 
that is present for the first-person perspective (Fajen and Warren, 
2003). 

PATH PERCEPTION 

In our analysis heading was tangent to the path; however, in 
general, this can differ. Instantaneous flow is ambiguous with 



respect to eye-rotations and or body/head-rotations. Given eye- 
rotations and a straight path trajectory, flows have a rotational 
component. For body-rotations the path trajectory is curvilin- 
ear and the flow has a rotational component too. The ambiguity 
in the instantaneous motion field yields to a strong bias with 
respect to the expectation about the path either being linear or 
curved as given, e.g., by the instruction as part of the exper- 
iment (Li and Warren, 2004). Only when gaze pointed along 
the heading direction (zero heading), path perception is accu- 
rate (Li and Cheng, 2011). The perception of curvilinear paths 
is improved by dense motion parallax cues and a reference 
object (Li and Warren, 2000). In our formulations for curvilin- 
ear path motion gaze and heading were aligned, which resolves 
the ambiguity. Several degrees of freedom e.g., eye-movements 
(Warren and Hannon, 1990; Royden et al., 1992, 1994), have 
been left out in our model to focus on data from Warren and 
Saunders (1995), Royden and Hildreth (1996), and Fajen and 
Kim (2002) and not to further complicate the model equations. 
Future work will focus on the interplay between eye-, head-, 
and body-movements during linear or curvilinear path motion 
and the various segmentation cues present in the generated 
flows. 
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